{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# News Article Bias Detection with Semantic Classification\n",
    "\n",
    "This notebook demonstrates how to use fenic's semantic classification capabilities to detect editorial bias and analyze news articles. We'll walk through:\n",
    "\n",
    "- Language Analysis using `semantic.extract()` to find biased, emotional, or sensationalist language.\n",
    "- Political Bias Classifcation using `semantic.classify()` grounded in the extracted data.\n",
    "- News Topic Classification using `semantic.classify()`\n",
    "- Merging the information together using `semantic.reduce()` to create a 'Media Profile' summary for each analyzed News Source\n",
    "\n",
    "This is a practical example of how semantic classification can provide insights into media content."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## Initial Setup\n",
    "\n",
    "First, let's configure our fenic session with semantic capabilities using an OpenAI model for our language processing tasks. Alternatively, uncomment the additional supplied configurations to use an Gemini or Anthropic model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fenic as fc\n",
    "from pydantic import BaseModel, Field\n",
    "from fenic import col, lit\n",
    "\n",
    "# Configure session with semantic capabilities\n",
    "print(\"🔧 Configuring fenic session...\")\n",
    "\n",
    "config = fc.SessionConfig(\n",
    "    app_name=\"news_analysis\",\n",
    "    semantic=fc.SemanticConfig(\n",
    "        language_models={ \n",
    "            \"openai\": fc.OpenAIModelConfig(\n",
    "                model_name=\"gpt-4o-mini\",\n",
    "                rpm=500,\n",
    "                tpm=200_000\n",
    "            ),\n",
    "            # \"anthropic\": fc.AnthropicModelConfig(\n",
    "            #     model_name=\"claude-sonnet-4-0\",\n",
    "            #     rpm=500,\n",
    "            #     input_tpm=80_000,\n",
    "            #     output_tpm=32_000,\n",
    "            # ),\n",
    "            #  \"gemini\": fc.GoogleGLAModelConfig(\n",
    "            #     model_name=\"gemini-2.0-flash\",\n",
    "            #     rpm=500,\n",
    "            #     tpm=1_000_000\n",
    "            # ),\n",
    "        }\n",
    "    )\n",
    ")\n",
    "\n",
    "# Create session\n",
    "session = fc.Session.get_or_create(config)\n",
    "print(\"✅ Session configured successfully!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample News Articles Dataset\n",
    "\n",
    "We'll work with a curated dataset of news articles from different sources covering the same stories. This allows us to analyze how different outlets report on identical events and detect bias patterns.\n",
    "\n",
    "Our dataset includes articles from:\n",
    "- **Neutral sources** (Global Wire Service, National Press Bureau)\n",
    "- **Left-leaning sources** (Progressive Voice, Social Justice Today)\n",
    "- **Right-leaning sources** (Liberty Herald, Free Market Weekly)\n",
    "- **Mixed sources** (Balanced Tribune, Independent Monitor)\n",
    "\n",
    "Each source has multiple articles to demonstrate consistency in bias patterns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sample news articles - multiple articles per source to show bias patterns\n",
    "news_articles = [\n",
    "    # Global Wire Service (Neutral source, Reuters-style) - 3 articles\n",
    "    {\n",
    "        \"source\": \"Global Wire Service\",\n",
    "        \"headline\": \"Federal Reserve Raises Interest Rates by 0.25 Percentage Points\",\n",
    "        \"content\": \"The Federal Reserve announced a quarter-point increase in interest rates Wednesday, bringing the federal funds rate to 5.5%. The decision was unanimous among voting members. Fed Chair Jerome Powell cited persistent inflation concerns and a robust labor market as key factors. The rate hike affects borrowing costs for consumers and businesses. Economic analysts had predicted the move following recent inflation data showing prices remained above the Fed's 2% target.\"\n",
    "    },\n",
    "    {\n",
    "        \"source\": \"Global Wire Service\",\n",
    "        \"headline\": \"OpenAI Launches GPT-4 Turbo with 128K Context Window\",\n",
    "        \"content\": \"OpenAI today announced GPT-4 Turbo, featuring a 128,000 token context window and updated training data through April 2024. The model offers improved instruction following and reduced likelihood of generating harmful content. Pricing is set at $0.01 per 1K input tokens and $0.03 per 1K output tokens. The release includes enhanced support for JSON mode and function calling. Developer early access begins this week, with general availability planned for December.\"\n",
    "    },\n",
    "    {\n",
    "        \"source\": \"Global Wire Service\",\n",
    "        \"headline\": \"Climate Summit Reaches Agreement on Fossil Fuel Transition\",\n",
    "        \"content\": \"Delegates at the COP28 climate summit in Dubai reached a consensus agreement calling for a transition away from fossil fuels in energy systems. The deal, approved by nearly 200 countries, marks the first time a COP agreement explicitly mentions fossil fuels. However, the agreement uses the phrase 'transitioning away' rather than 'phasing out,' reflecting compromises necessary to secure broad support. Environmental groups expressed mixed reactions, with some praising the historic mention while others criticized the lack of binding timelines.\"\n",
    "    },\n",
    "    \n",
    "    # Progressive Voice (Left-leaning source) - 3 articles  \n",
    "    {\n",
    "        \"source\": \"Progressive Voice\",\n",
    "        \"headline\": \"Fed's Rate Hike Threatens Working Families as Corporate Profits Soar\",\n",
    "        \"content\": \"Once again, the Federal Reserve has chosen to burden working families with higher borrowing costs while Wall Street celebrates record profits. Wednesday's rate hike to 5.5% will make mortgages, credit cards, and student loans more expensive for millions of Americans already struggling with housing costs. Meanwhile, corporate executives continue awarding themselves massive bonuses. This regressive monetary policy prioritizes the wealthy elite over middle-class families who desperately need relief.\"\n",
    "    },\n",
    "    {\n",
    "        \"source\": \"Progressive Voice\", \n",
    "        \"headline\": \"Big Tech's AI Surveillance Threatens Democratic Values\",\n",
    "        \"content\": \"OpenAI's latest AI release represents another troubling escalation in Silicon Valley's surveillance capitalism model. These systems hoover up personal data and creative content without meaningful consent from users. Artists, writers, and creators see their work exploited to train AI systems that directly compete with human creativity. Meanwhile, users surrender intimate conversations to corporate servers with little transparency. We need immediate regulation to protect digital rights and prevent tech giants from privatizing human knowledge for profit.\"\n",
    "    },\n",
    "    {\n",
    "        \"source\": \"Progressive Voice\",\n",
    "        \"headline\": \"Climate Summit's Weak Language Betrays Future Generations\", \n",
    "        \"content\": \"The COP28 agreement represents a devastating failure to confront the climate emergency with the urgency science demands. By choosing vague 'transition' language over concrete 'phase out' commitments, world leaders have once again capitulated to fossil fuel lobbying and corporate interests. Young climate activists who traveled to Dubai seeking real action have been betrayed by politicians who prioritize industry profits over planetary survival. We cannot afford more empty promises while the climate crisis accelerates.\"\n",
    "    },\n",
    "    \n",
    "    # Liberty Herald (Right-leaning source) - 3 articles\n",
    "    {\n",
    "        \"source\": \"Liberty Herald\",\n",
    "        \"headline\": \"Fed's Prudent Rate Decision Reinforces Economic Stability\",\n",
    "        \"content\": \"The Federal Reserve's measured quarter-point rate increase demonstrates responsible monetary policy that will preserve long-term economic prosperity. By raising rates to 5.5%, Fed officials are taking necessary steps to prevent runaway inflation that would devastate savings and fixed incomes. This disciplined approach protects the purchasing power that American families have worked hard to build. Free market principles and sound fiscal management require tough decisions that ensure sustainable growth for job creators and investors.\"\n",
    "    },\n",
    "    {\n",
    "        \"source\": \"Liberty Herald\",\n",
    "        \"headline\": \"American AI Innovation Leads Global Technology Revolution\",\n",
    "        \"content\": \"OpenAI's breakthrough demonstrates why American innovation continues to lead the world in transformative technology. This achievement showcases the power of free enterprise and competitive markets to deliver solutions that benefit humanity. While other nations impose heavy-handed regulations that stifle innovation, American companies are unleashing AI capabilities that will create jobs, boost productivity, and solve complex problems. America's technological superiority depends on supporting pioneering companies through pro-growth policies and reduced government interference.\"\n",
    "    },\n",
    "    {\n",
    "        \"source\": \"Liberty Herald\",\n",
    "        \"headline\": \"Pragmatic Climate Deal Balances Environmental Goals with Economic Reality\",\n",
    "        \"content\": \"The COP28 agreement demonstrates mature leadership by acknowledging environmental concerns while protecting economic stability and energy security. The careful 'transition away' language recognizes that abrupt fossil fuel elimination would devastate working families and developing nations that depend on affordable energy. American energy producers have already reduced emissions through innovation and cleaner technologies, proving that market solutions work better than government mandates. This balanced approach protects jobs while investing in alternatives.\"\n",
    "    }\n",
    "]\n",
    "\n",
    "print(f\"📰 Loaded {len(news_articles)} news articles from various sources\")\n",
    "print(f\"🔍 Sources: {len(set(article['source'] for article in news_articles))} unique news outlets\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create DataFrame and Dataset Overview\n",
    "\n",
    "Let's convert our news articles into a fenic DataFrame and examine the composition of our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create DataFrame from news articles\n",
    "df = session.create_dataframe(news_articles)\n",
    "\n",
    "print(\"📰 News Bias Detection Pipeline\")\n",
    "print(\"=\" * 70)\n",
    "print(f\"Analyzing {df.count()} news articles from {df.select('source').drop_duplicates(['source']).count()} sources\")\n",
    "\n",
    "# Show dataset composition\n",
    "print(\"\\n📊 Dataset Composition:\")\n",
    "source_counts = df.group_by(\"source\").agg(fc.count(\"*\").alias(\"articles\")).order_by(\"source\")\n",
    "source_counts.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define Analysis Schema\n",
    "\n",
    "Before we perform semantic operations, let's define a Pydantic model that will help us extract structured information about bias indicators, emotional language, and opinion markers from each article."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define Pydantic model for detailed article analysis\n",
    "class ArticleAnalysis(BaseModel):\n",
    "    \"\"\"Comprehensive analysis of news article content and bias\"\"\"\n",
    "    bias_indicators: str = Field(..., description=\"Key words or phrases that indicate political bias\")\n",
    "    emotional_language: str = Field(..., description=\"Emotionally charged words or neutral descriptive language\")\n",
    "    opinion_markers: str = Field(..., description=\"Words or phrases that signal opinion vs. factual reporting\")\n",
    "\n",
    "print(\"✅ Analysis schema defined - ready for semantic extraction!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 1: Content Preprocessing and Information Extraction\n",
    "\n",
    "First, we'll combine headlines and content for richer context, then extract key information about bias indicators, emotional language, and opinion markers from each article. This sets up our data for the classification stage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"🔍 Performing semantic bias detection...\")\n",
    "print(\"First, we extract key information from each article.\\n\")\n",
    "\n",
    "# Create combined text for context-aware analysis\n",
    "combined_content = fc.text.concat(\n",
    "    fc.col(\"headline\"), \n",
    "    fc.lit(\" | \"), \n",
    "    fc.col(\"content\")\n",
    ")\n",
    "\n",
    "# Extract information and classify topics\n",
    "# We can use `.cache()` to ensure these expensive LLM operations don't need to be re-run each time we modify\n",
    "# the resultant materialized dataframe.\n",
    "enriched_df = df.with_column(\"combined_content\", combined_content).select(\n",
    "    fc.col(\"source\"),\n",
    "    fc.col(\"headline\"),\n",
    "    fc.col(\"content\"),\n",
    "    # Primary topic classification\n",
    "    fc.semantic.classify(\n",
    "        fc.col(\"combined_content\"),\n",
    "        [\"politics\", \"technology\", \"business\", \"climate\", \"healthcare\"]\n",
    "    ).alias(\"primary_topic\"),\n",
    "    # Content Metadata using semantic.extract\n",
    "    fc.semantic.extract(\n",
    "        fc.col(\"combined_content\"),\n",
    "        ArticleAnalysis,\n",
    "        max_output_tokens=512,\n",
    "    ).alias(\"analysis_metadata\"),\n",
    ").unnest(\"analysis_metadata\").cache()\n",
    "enriched_df.collect()\n",
    "print(\"✅ Information extraction completed!\")\n",
    "print(\"\\n📊 Sample extracted information:\")\n",
    "enriched_df.select(\"source\", \"headline\", \"primary_topic\", \"bias_indicators\", \"emotional_language\", \"opinion_markers\").show(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 2: Political Bias Classification\n",
    "\n",
    "Now we'll use the extracted information to classify the political bias of each article. We combine the topic, bias indicators, emotional language, and opinion markers to give the model rich context for accurate bias detection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "# Combine extracted information for bias classification\n",
    "combined_extracts = fc.text.concat(\n",
    "    lit(\"Primary Topic: \"),\n",
    "    fc.col(\"primary_topic\"),\n",
    "    lit(\"Political Bias Indicators: \"),\n",
    "    fc.col(\"bias_indicators\"),\n",
    "    lit(\"||||||||||||\"),\n",
    "    lit(\"Emotional Language Summary: \"),\n",
    "    fc.col(\"emotional_language\"),\n",
    "    lit(\"||||||||||||\"),\n",
    "    lit(\"Opinion Markers: \"),\n",
    "    fc.col(\"opinion_markers\")\n",
    ")\n",
    "\n",
    "enriched_df = enriched_df.with_column(\"combined_extracts\", combined_extracts)\n",
    "\n",
    "# Classify political bias and journalistic style\n",
    "results_df = enriched_df.select(\n",
    "    \"*\",\n",
    "    fc.semantic.classify(\n",
    "        col(\"combined_extracts\"), \n",
    "        [\"far_left\", \"left_leaning\", \"neutral\", \"right_leaning\", \"far_right\"]\n",
    "    ).alias(\"content_bias\"),\n",
    "    fc.semantic.classify(\n",
    "        col(\"combined_extracts\"), \n",
    "        [\"sensationalist\", \"informational\"]\n",
    "    ).alias(\"journalistic_style\")\n",
    ").cache()\n",
    "results_df.collect()\n",
    "print(\"✅ Bias classification completed!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results: Complete Bias Detection Analysis\n",
    "\n",
    "Let's examine our complete results, showing how each article was classified for topic, bias, and journalistic style."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"📊 Complete Bias Detection Results:\")\n",
    "print(\"=\" * 70)\n",
    "\n",
    "# Show key results for each article\n",
    "summary_results = results_df.select(\n",
    "    \"source\",\n",
    "    \"headline\",\n",
    "    \"primary_topic\",\n",
    "    \"content_bias\",\n",
    "    \"journalistic_style\"\n",
    ")\n",
    "summary_results.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bias Language Analysis\n",
    "\n",
    "Let's examine the specific language patterns that indicate bias versus neutral reporting. This helps us understand what linguistic markers the model identified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Bias Indicators Analysis\n",
    "bias_indicators_df = results_df.select(\n",
    "    \"source\",\n",
    "    \"headline\",\n",
    "    \"content_bias\",\n",
    "    \"bias_indicators\",\n",
    "    \"emotional_language\",\n",
    "    \"opinion_markers\"\n",
    ")\n",
    "\n",
    "print(\"🔍 Bias Language Analysis:\")\n",
    "print(\"=\" * 70)\n",
    "\n",
    "# Show examples of neutral vs biased language\n",
    "print(\"📰 Neutral Articles - Language Patterns:\")\n",
    "neutral_examples = bias_indicators_df.filter(\n",
    "    fc.col(\"content_bias\") == \"neutral\"\n",
    ").select(\"source\", \"headline\", \"bias_indicators\", \"opinion_markers\")\n",
    "neutral_examples.show(5)\n",
    "\n",
    "print(\"\\n📰 Biased Articles - Language Patterns:\")\n",
    "biased_examples = bias_indicators_df.filter(\n",
    "    (fc.col(\"content_bias\") != \"neutral\")\n",
    ").select(\"source\", \"headline\", \"content_bias\", \"bias_indicators\", \"emotional_language\", \"opinion_markers\")\n",
    "biased_examples.show(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AI-Generated Media Profiles\n",
    "\n",
    "Finally, let's use fenic's semantic reduction capabilities to generate media profiles for each news source based on all the information we've extracted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate semantic summaries of language patterns for each source\n",
    "source_language_profiles = results_df.group_by(\"source\").agg(\n",
    "    # Use semantic.reduce to produce a media profile for each source, without including the entire original articles.\n",
    "    fc.semantic.reduce(\n",
    "        \"\"\"\n",
    "           Create a concise (3-5 sentence) media profile for {source} based on the following information we have extracted from its articles:\n",
    "           Detected Political Bias: {content_bias}\n",
    "           Detected Bias Indicators: {bias_indicators}\n",
    "           Opinion Indicators: {opinion_markers}\n",
    "           Emotional Language: {emotional_language}\n",
    "           Journalistic Style: {journalistic_style}\n",
    "           \n",
    "           Summarize the information provided, limit your use of direct quotes from the text.\n",
    "        \"\"\",\n",
    "        max_output_tokens=512,\n",
    "    ).alias(\"source_profile\"),\n",
    ").select(col(\"source\"), col(\"source_profile\"))\n",
    "\n",
    "print(\"🏢 AI-Generated Media Profiles:\")\n",
    "print(\"-\" * 50)\n",
    "source_language_profiles.show()\n",
    "\n",
    "# Clean up session\n",
    "session.stop()\n",
    "\n",
    "print(\"\\n✅ News Bias Detection Complete!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Key Insights and Applications\n",
    "\n",
    "This analysis demonstrates several powerful capabilities of semantic classification:\n",
    "\n",
    "### 🎯 Key Insights Demonstrated:\n",
    "- **Content-based bias detection** without relying on source name predictions\n",
    "- **Source consistency analysis** across multiple articles\n",
    "- **Language pattern identification** for bias indicators\n",
    "- **Topic-agnostic bias detection** (same source biased across different topics)\n",
    "- **Quality assessment** with confidence scoring\n",
    "\n",
    "### 🔍 Practical Applications:\n",
    "- **Media literacy education** showing how bias manifests in language\n",
    "- **Content moderation** for balanced information presentation\n",
    "- **News aggregation** with bias awareness\n",
    "- **Research on editorial patterns** and media analysis\n",
    "\n",
    "### 🚀 Next Steps:\n",
    "- Try analyzing your own news articles or text data\n",
    "- Experiment with different classification categories\n",
    "- Combine with other semantic operations like extraction and mapping\n",
    "- Build automated content analysis pipelines"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
